Fixes #27950: [Datalake] JSON columns incorrectly typed as STRING for empty dict values by mohittilala · Pull Request #27951 · open-metadata/OpenMetadata

mohittilala · 2026-05-07T03:17:18Z

Describe your changes:

Fixes #27950

Changes in OpenMetadata submodule (datalake_utils.py):

Empty dict/list columns now correctly typed as JSON/ARRAY instead of STRING
Skip ast.literal_eval round-trip for already-parsed dict/list values
get_children handles parsed dicts and JSON strings independently — no more TypeError log spam

Tests added (tests/unit/utils/test_datalake.py):

Unit tests for fetch_col_types and get_children with parsed objects, empty containers, mixed types
End-to-end tests reading real fixture files through the full _read_json_object → _get_columns pipeline

Type of change:

Checklist:

I have read the CONTRIBUTING document.
My PR title is Fixes <issue-number>: <short explanation>
I have commented on my code, particularly in hard-to-understand areas.
For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

Bug fix

I have added a test that covers the exact scenario we are fixing. For complex issues, comment the issue number in the test for future reference.

Summary by Gitar

Type detection logic:
- Replaced lexicographic max() type resolution with _TYPE_PRECEDENCE mapping in fetch_col_types.
- Prevents structured types (dict, list) from being incorrectly downgraded to STRING in mixed-type columns.
Testing additions:
- Added TestFetchColTypesMixedTypes to verify correct resolution for mixed dict/str, list/str, and numeric column types.

_{This will update automatically on new commits.}

… values

Copilot

Pull request overview

Fixes an ingestion bug in the Datalake connector where JSON-like columns (especially empty {} / [] values coming from single-object JSON files) were incorrectly inferred as STRING, and where parsing children could emit repeated TypeError debug logs.

Changes:

Update column type inference to treat non-null object columns as candidates even when values are falsy containers, and avoid unnecessary ast.literal_eval for already-parsed dict/list values.
Rework JSON children extraction to handle mixed parsed-dict and JSON-string values without TypeError noise.
Add unit + fixture-based tests covering parsed objects, empty containers, and single-object JSON ingestion behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
`ingestion/src/metadata/utils/datalake/datalake_utils.py`	Fixes type inference for empty dict/list values and makes `get_children` robust to parsed objects vs JSON strings.
`ingestion/tests/unit/utils/test_datalake.py`	Adds targeted tests for `fetch_col_types`/`get_children` and fixture-driven single-object JSON parsing.
`ingestion/tests/unit/resources/datalake/dbt_manifest.json`	Adds a representative single-object dbt manifest fixture with multiple empty-object fields.
`ingestion/tests/unit/resources/datalake/dbt_catalog.json`	Adds a representative single-object dbt catalog fixture with nested dicts and nulls.

…fy type checker

github-actions · 2026-05-07T05:56:52Z

🟡 Playwright Results — all passed (21 flaky)

✅ 4064 passed · ❌ 0 failed · 🟡 21 flaky · ⏭️ 86 skipped

Shard	Passed	Flaky	Skipped
🟡 Shard 1	297	2	4
🟡 Shard 2	757	11	8
🟡 Shard 3	780	1	7
🟡 Shard 4	786	4	18
✅ Shard 5	709	0	41
🟡 Shard 6	735	3	8

🟡 21 flaky test(s) (passed on retry)

Pages/AuditLogs.spec.ts › should create audit log entry when glossary is soft deleted (shard 1, 1 retry)
Pages/UserCreationWithPersona.spec.ts › Create user with persona and verify on profile (shard 1, 1 retry)
Features/ActivityAPI.spec.ts › Activity event shows the actor who made the change (shard 2, 1 retry)
Features/BulkEditEntity.spec.ts › Glossary (shard 2, 1 retry)
Features/Glossary/GlossaryWorkflow.spec.ts › should display correct status badge color and icon (shard 2, 1 retry)
Features/Glossary/GlossaryWorkflow.spec.ts › should start term as Draft when glossary has reviewers (shard 2, 1 retry)
Features/IncidentManager.spec.ts › Complete Incident lifecycle with table owner (shard 2, 2 retries)
Features/IncidentManager.spec.ts › Next, Previous and page indicator (shard 2, 2 retries)
Features/KnowledgeCenter.spec.ts › Article mentions in description should working for Knowledge Center (shard 2, 1 retry)
Features/KnowledgeCenterList.spec.ts › Knowledge Center List - Test add article button (shard 2, 1 retry)
Features/KnowledgeCenterTextEditor.spec.ts › Rich Text Editor - Text Formatting (shard 2, 1 retry)
Features/KnowledgeCenterTextEditor.spec.ts › Rich Text Editor - Text Formatting (shard 2, 1 retry)
Features/KnowledgeCenterTextEditor.spec.ts › Rich Text Editor - Text Formatting (shard 2, 1 retry)
Features/RTL.spec.ts › Verify Following widget functionality (shard 3, 1 retry)
Pages/CustomProperties.spec.ts › Create custom property and configure search for Dashboard (shard 4, 1 retry)
Pages/CustomProperties.spec.ts › Should verify property name is visible for apiCollection in right panel (shard 4, 1 retry)
Pages/DataContracts.spec.ts › Create Data Contract and validate for Pipeline (shard 4, 1 retry)
Pages/DataProducts.spec.ts › Search Data Products (shard 4, 1 retry)
Pages/ExplorePageRightPanel.spec.ts › Should allow Data Consumer to edit glossary terms for searchIndex (shard 6, 1 retry)
Pages/Glossary.spec.ts › Column dropdown drag-and-drop functionality for Glossary Terms table (shard 6, 1 retry)
Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 6, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

… utils

…l_types

gitar-bot · 2026-05-11T08:57:18Z

Code Review ✅ Approved

Replaces lexicographic type resolution with explicit precedence in datalake utils to correctly identify JSON and array columns. Added comprehensive unit tests to ensure accurate type detection for empty containers and mixed-type inputs.

Options

Display: compact → Showing less information.

Comment with these commands to change:

`Compact`
`gitar display:verbose`

_{Was this helpful? React with 👍 / 👎 | Gitar}

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

sonarqubecloud · 2026-05-11T10:07:42Z

Quality Gate passed for 'open-metadata-ingestion'

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
86.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions · 2026-05-11T12:33:31Z

Failed to cherry-pick changes to the 1.12.7 branch.
Please cherry-pick the changes manually.
You can find more details here.

github-actions · 2026-05-11T12:34:50Z

Failed to cherry-pick changes to the 1.13 branch.
Please cherry-pick the changes manually.
You can find more details here.

fix: datalake JSON columns incorrectly typed as STRING for empty dict…

c14e05b

… values

mohittilala self-assigned this May 7, 2026

Copilot AI review requested due to automatic review settings May 7, 2026 03:17

mohittilala requested a review from a team as a code owner May 7, 2026 03:17

mohittilala added bug Something isn't working Ingestion safe to test Add this label to run secure Github workflows on PRs To release Will cherry-pick this PR into the release branch python Pull requests that update python code labels May 7, 2026

Copilot started reviewing on behalf of mohittilala May 7, 2026 03:17 View session

Copilot AI reviewed May 7, 2026

View reviewed changes

Comment thread ingestion/src/metadata/utils/datalake/datalake_utils.py Outdated

Comment thread ingestion/tests/unit/utils/test_datalake.py

mohittilala had a problem deploying to test May 7, 2026 03:28 — with GitHub Actions Failure

mohittilala temporarily deployed to test May 7, 2026 03:28 — with GitHub Actions Inactive

fix: wrap df_row_val with str() for strptime and parse calls to satis…

f17df88

…fy type checker

fix: address static check type errors and review comments in datalake…

afd7898

… utils

mohittilala temporarily deployed to test May 7, 2026 06:58 — with GitHub Actions Inactive

mohittilala had a problem deploying to test May 7, 2026 06:58 — with GitHub Actions Failure

mohittilala temporarily deployed to test May 7, 2026 06:58 — with GitHub Actions Inactive

mohittilala temporarily deployed to test May 7, 2026 13:24 — with GitHub Actions Inactive

Restore debug logging, fix dead-code fallback, strengthen tests

a0f0f01

Copilot AI review requested due to automatic review settings May 8, 2026 04:50

Copilot started reviewing on behalf of mohittilala May 8, 2026 04:51 View session

mohittilala temporarily deployed to test May 8, 2026 05:00 — with GitHub Actions Inactive

mohittilala temporarily deployed to test May 8, 2026 07:22 — with GitHub Actions Inactive

TeddyCr approved these changes May 8, 2026

View reviewed changes

TeddyCr previously approved these changes May 11, 2026

View reviewed changes

Replace lexicographic max() with explicit type precedence in fetch_co…

6ab38c8

…l_types

mohittilala dismissed TeddyCr’s stale review via 6ab38c8 May 11, 2026 05:10

mohittilala temporarily deployed to test May 11, 2026 05:22 — with GitHub Actions Inactive

Merge branch 'main' into fix/datalake-json-column-type-detection

942925e

Copilot AI review requested due to automatic review settings May 11, 2026 08:56

Copilot started reviewing on behalf of mohittilala May 11, 2026 08:57 View session

Copilot AI reviewed May 11, 2026

View reviewed changes

Comment thread ingestion/tests/unit/utils/test_datalake.py

mohittilala temporarily deployed to test May 11, 2026 09:06 — with GitHub Actions Inactive

ulixius9 approved these changes May 11, 2026

View reviewed changes

ulixius9 merged commit 3d6fd71 into main May 11, 2026
64 of 66 checks passed

ulixius9 deleted the fix/datalake-json-column-type-detection branch May 11, 2026 12:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes #27950: [Datalake] JSON columns incorrectly typed as STRING for empty dict values#27951

Fixes #27950: [Datalake] JSON columns incorrectly typed as STRING for empty dict values#27951
ulixius9 merged 6 commits into
mainfrom
fix/datalake-json-column-type-detection

mohittilala commented May 7, 2026 •

edited by gitar-bot Bot

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 7, 2026 •

edited

Loading

Uh oh!

gitar-bot Bot commented May 11, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

sonarqubecloud Bot commented May 11, 2026

Uh oh!

Uh oh!

github-actions Bot commented May 11, 2026

Uh oh!

github-actions Bot commented May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

mohittilala commented May 7, 2026 • edited by gitar-bot Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes:

Type of change:

Checklist:

Summary by Gitar

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟡 Playwright Results — all passed (21 flaky)

Uh oh!

gitar-bot Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

sonarqubecloud Bot commented May 11, 2026

Quality Gate passed for 'open-metadata-ingestion'

Uh oh!

Uh oh!

github-actions Bot commented May 11, 2026

Uh oh!

github-actions Bot commented May 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mohittilala commented May 7, 2026 •

edited by gitar-bot Bot

Loading

github-actions Bot commented May 7, 2026 •

edited

Loading

gitar-bot Bot commented May 11, 2026 •

edited

Loading